Effective error recovery strategies for multimodal form-filling applications

The goal of the research described in this article is to determine in what way speech recognition errors can be handled best in a multimodal form-filling interface. Besides two well-known error correction mechanisms (re-speaking the value and choosing the correct value from a list of alternatives), the interface offers a novel correction mechanism in which the user selects the first letter of the target word from a soft-keyboard, after which the utterance is recognized once again, with a limited language model and lexicon. The multimodal interface that was used is a web-based form-filling GUI, extended with a speech overlay, which allows for pen and speech input. The effectiveness and efficiency of the error correction mechanisms, the error correction strategies that are applied by the users and the effects on user satisfaction were studied in an evaluation in which the interface was tested in two conditions: in one condition (LIST), the interface provides only re-speaking and the alternatives list as error correction facilities. In the other condition (LETTER), the interface provides the soft-keyboard technique as an additional error correction facility. The study shows that error correction was more effective in the LETTER condition than in the LIST condition. The Keyboard correction facility enables the users to solve errors that could not be solved using the Re-speak method or by choosing from a list of alternatives. In spite of its low effectiveness, subjects initially attempted to use Re-speaking for error correction in both interfaces. However, we also found that subjects rapidly learned to choose the most effective option (Keyboard) immediately as they gain experience. The user satisfaction turned out to be higher for the LETTER interface than for the LIST interface: subjects considered the LETTER interface to be more useful and less frustrating and they felt more in control. As a result, most subjects clearly preferred the LETTER interface. 2005 Elsevier B.V. All rights reserved.